The Infiniband® network includes nodes that communicate through a channel-based switched fabric (Infiniband® is a registered trademark of Infiniband Trade Association, located in Beaverton, Oreg.). For example, the nodes may be a host, an input/output subsystem, or a router which connects to another network. The switched fabric is made up of a collection of switches, routers, and/or links that connect a set of channel adapters. The channel adapters form an interface between the switched fabric and the nodes. The channel adapter of the host is referred to as a host channel adapter. The channel adapter of an I/O subsystem is referred to as a target channel adapter.
In Infiniband®, two processes communicate using a queue pair. A queue pair includes a send queue and a receive queue. Specifically, in order for a process to send a message to another process, the process posts the message to the send queue. The host channel adapter sends the message in the form of packets to the channel adapter having the receive queue. Each packet that is sent may include a packet sequence number. Logic associated with the receive queue ensures that packets are processed in a particular order using the packet sequence number.
Infiniband® supports operations such as remote direct memory access (RDMA) read and write operations and reliable connections. Specifically, a requesting process may send a request in the message for the receiver to store data and/or return data. In response, the remote node's channel adapter includes functionality to store and/or return the requested data to the requesting process.
In network communications, reliable connections (both for remote copying and extended remote copying) are implemented by the requester having an acknowledgement when a packet is successfully transmitted or having a timeout if an acknowledgement is not received within a fixed programmable time after a packet is sent. When an acknowledgement is received by the requestor, the requestor must ‘complete’ the request by writing required data to a completion queue in main memory. The data for the completion is obtained from the send queue of the corresponding queue pair (QP). A completion is written to the completion queue in order to provide a user with information about the send request. In the event that packet transmission is not successful, after the timeout period has elapsed, the initial transmission is followed by packet retransmission, where duplicated packets are ignored by the responder. Once a timeout for a given request packet is detected, the requester may retry the request.